all right uh here let me put the title up um cool uh I'm really glad I followed that talk because this is um this is a talk about how we do uh decentralized indexing in um uh stature like it's a whole bunch of like backstory and stuff that you probably don't need to know but is kind of interesting and I thought you might like to hear about it anyway um but it's basically how we use ipni or we're going to use hopefully IP Andi to decentralize uh storage Network so just for context I don't expect you to know anything about stor it's brand new but it's kind of the evolution of web free storage uh becoming a completely decentralized storage Network um for both storage and retrieval um woo okay so from the top uh it's all about like transferring tags from one place to another uh so we're talking about taking files uh do something to them make a uh dag directed ack graph uh and then basically try and put them on a node in network uh somewhere so uh yeah how how to do that so there's kind of two approaches to this basically there's like the traditional uh kind of well traditional in ipfs sense pull based approach or you can do like pushing as well so uh you've got like your local thing on the left hand side what you're doing on the client and the remote place where you actually want stuff stored so that it can be retrieved even when you shut your laptop there's still node out there that's actually hosting your stuff and sending it to off to other people so um traditionally pole based approach create your dag put it in a local ipfs node right and then you uh you grab hold of that root hash your your baffy uh and you basically say hey remote pull this baffy and the and the remote uh node will eventually uh hopefully because it's built on top of lip p2b um do magic and uh and pull uh pull that whole dag from your local node into the remote node um and that that works and that's uh that's super cool uh and I I really like it but it there's certain times and instances and places where it doesn't always work um and uh let me just explain a little bit um sometimes there's uh a wall that's on fire uh there like firewalls there's gats there's like General connectivity uh issues and stuff like that uh that mean that Li ptb is not always able to to connect from one place to another it's really good it's really good at that but it's uh not 100% all the time uh good unfortunately um the other reason is that like um you might not want to be running an ipfs node like you might be on a mobile uh you could be running a Raspberry Pi and you don't have enough resources to be running and whole ipfs node you might like create this dag and be like I don't actually want to store this on my ipfs node I just want to send it up there for the web to get um uh yeah so lots of reasons why you might might not want to um also it could another reason just for a bit um you could have like an app uh that's that's like maybe you're minting nfts or something you're not it's maybe it's Overkill to run an ipfs node just to add stuff and and get it um get it moved to somewhere else where it can be fetched forever uh but I think the biggest thing for me is that there's there's no like transactional guarantee like once you ask this remote node to uh to pull up this uh pull this dag from your local one you you can't like you don't know when it's done um if you've got a big file it could take a long time I can't just shut my laptop and wander off uh ever cuz like it might not happen at all uh and so like how do I know there's no like call back that I can get that that will tell me when that when it's finished when it's done um that's really annoying so um in some instances it's um it's better to do like a a kind of more push based thing where uh you especially in like uh the services that that that I've been trying try run for a while now uh on the client side instead of putting that dag that you've created into a local ipfs node maybe you pack it into a a car file car file is content addressed archived I um this I will depict that as a car Emoji in the slides from now on but just it's not a real car it's a file okay just you know you get it um so anyway put your dag in a car and once you've got a car file um what you can do is you can just do like a regular old HTTP put to an ipfs instance and um and we used to put to Cluster IPS cluster um uh because it support it has an API which allows you to just put um car files to it which is which is pretty rad um uh so that's that's super easy and once you've put that file you'll know that it has got there because you did the request and then you know it succeeded so uh there's there's that sort of more more transactional guarantee there um once cluster receives it it will put it to multiple nodes because cluster uh but uh what it does is it it will take that car file unpack it uh and for every block that it finds in that car file it will actually put a block in its uh Block store The Block store is um as you can imagine indexed by uh the the hash of the block uh so when ipfs uh cluster needs to serve that back to someone it can just uh you know it'll get it'll get asked for a CID and it can go to The Block store and grab it grab that data and send it back so um yeah IBS cluster is amazing but there is a non-zero management overhead if you're running cluster you are running multiple ipfs nodes uh potentially on like bare metal hardware if you depending on how much you're storing uh you're going to have to have quite big discs you uh you need to do things like uh update software occasionally sometimes well okay it's maybe uh Kubo is more a little bit more stable nowadays but um when I was experiencing this this um the the like Kubo would just occasionally get tired and fall over and and then you'd have to go and sort of um restart it so there's like also like pager Duty and stuff um which is kind of frustrating um uh and and then like like you've also got R uh cluster uh being like right target if it gets some data that's super busy then uh if it's that's super popular it's also the read Target and it's also doing a whole lunch bunch of like DHD puts it can get busy uh and uh and yeah so there is there is something to consider there it's a it's ipfs it's a complicated system that runs and you have to uh babysit it essentially so um so having experienced this um at quite a large scale for a long time like we were running like a cluster of 52 nodes or something uh at one point and um the this idea of like cars at rest came around where we were like why don't we not do that and why we uh why don't we not do the cluster thing why don't we just put cars to like some sort of bucket somewhere some simple storage uh and then and not do anything with them uh and so instead of like uh sending it to Cluster we' just send the car and it would persist as a car uh on some disc somewhere uh just as it is um so that idea came around um and so that's kind of cool because you don't need to then take that car and unpack it um you've got like to the to on the dumb servers uh thing uh I'm interested in Dumb servers as well uh you don't you you basically have a a server that would uh that um can receive data take some bites in put it to disk and uh and also send bites back and that's all it does and it's uh and it doesn't really fall over there's nothing really to go wrong there because it's so simple uh which is great um but like content is still content addressed is it are we still good here and it's like yeah kind of like the car in and of itself is bunch of blocks which are all content addressed data anyway and if you take that car and kind of consider it as a one big block anyway you can actually take the hash of the car file as a whole um and and you can uh you can use that hash a lot of a lot of the places that allow you to just put bites also allow you to specify like you spec specify like the the hash like sh 2256 of the file that you're about to put so when the file arrives it can verify that that what you said you'd send is what it received and has been stored um so that's kind of cool um kind of like that uh but how do we how do we even serve this now because we don't have like a block store that we can kind of pick stuff out so um so uh we need to do something about that when you consider a car file is basically like a bunch of blocks that are um that are concatenated together like this five blocks here my Buffy Buffy ABCDE e um you can kind of you can kind of envisage the like you can just like create this information about the blocks that are in the car file like the offset and the their lengths for every single block and then if you need to grab a block then you can just sort of go to that bite that that offset and take that length and then you've got your block you just read it out of the file from those offsets um uh and and and so that's an index it's index over the car file um car V2 there's a spec uh that has indexes you can see the indexes are at the end there uh like there's a format there's two formats actually um uh and and so that's that that's interesting they're they're right at the end there of a car a car r like a car wrapped in a in a another thing um but anyway carv 2 has them but there's there's a couple of issues uh that are relevant to me at least um number one is where is the index um like I don't where where does where do I start reading the index do not know have to read the front of the file read the car header and not even like a known quantity of header uh to to to know exactly where the indexes I think is is how that works um so that's like two if you just got this simple sto storage thing just kind of two requests it's like read the header read the actual index and that's kind of annoying right um number two is um how how big are the blocks the in the index the format for the index is like all of the multi hases that are in the car file and their offsets but not their um not their lengths as well so if you want to read a block out of the car file you actually have to go like you take the offset read a bit because actually the offset is not to the block data it's to like every block has a little block header which tells you how long the block is so you have to do like a a a read to read how much you want to read from the car so it's like okay and then like and then you get you basically have to guess because you'll be like well okay my blocks are probably not going to be like more than two megabytes I'll ask for two megabytes and then and then uh and then see what I get or you say I'll just have all of it and then like whoever's giving it to you is going to egress probably more than you consume and you're going to you know all all sorts of inefficiencies and stuff there said um if we kind of put that side uh then it's kind of still kind of useful because we have like for every car we have a little file which is like all of the blocks in the car we don't really have like a giant index of everything it's just a a little a little index that you can use to pick out all of the data that you need from any given um any given car uh on a per car level um which is great and then it's like the missing Point um the missing thing that we H need here is just that we need like a mapping from our root hash to the car or cars that you can find that um that data in so so yeah index mapping those two things um so how does so how does it kind of work um you get like a request at the Gateway and uh it asks for a route s the Gateway is going to try and obviously get all the blocks to satisfy that request so it needs to consult that mapping thing that second second thing um and it knows then which cars it can find the dag in um and need to the index and then once it has read the index it knows where every single block that could possibly be be needed to satisfy this request is and it's localized to that the like the the the car that you're the dag that you're reading the car that you're reading um which is which is kind of cool um and you just export the blocks and send them send them back so um reading is super simple as well right um there's two bits of information here that are kind of um important that are like in um the existing kind of web free storage world uh we what we were doing was kind of generating these two bits of information sort of storing it on behalf of the user uh and that kind of that kind of sucks so we in in like decentralizing the whole of uh web free storage into this theature network um these two things really need to be addressed in some way they like they need to be decentralized uh and so what we want to do is decentralize all of the things um uh and this is where the uh ipni comes in which is why I'm glad my talk followed the like previous one um so yeah ipni um you just heard about it so I I don't can probably skip over what what it is but indexing indexing for the network um like at at a um kind of high level you can think of I as like a mapping from uh block hashes you can see in the URL here you can just look up any any hash um and get back uh like a provider uh some information about who who you can talk to to get gra grab hold of that um that particular block um you'll see here though that there's a little metadata field uh which is great um uh and you can basically put any anything in here uh within a reasonable uh reasonable size I I believe um and you can kind of think of like actually ipni is really just a mapping of like block hashes to arbitrary data and the provider information is just part of that arbitary data when you so IP IP publishing looks a little bit like this you publish an advert uh which is like advertisement whatever you want um this is like one of many um uh there's a whole chain of them and uh you basically have a the metadata is in the advertisement and the advertisement link links off to a list of hashes uh that that you are that you are publishing so when you come to weary for a particular hash you get back a list of advertisements a list of metadata for a given hash and you can have more than one uh so yeah so what we want to do is leverage that metadata to be able to store that information more information about um about the the uh the the SE or the car that we're we're working with that is not just like a provider record we've got we've got things like we want to store like you know the actual um offsets and lengths for example so there's two things that we want to store in the metadata like the hash of an index like a special special index we'll talk about in a sec um and uh the second thing is like a URL where that car was uploaded so like literally just a just wherever it ended up and that is just a dumb server somewhere in the decentralized network that you can make bite range requests to basically uh to extract blocks uh the index this is the special the special index that I talked about um it's it's it's a little bit better than the Carvey 2 index in my opinion for my use case uh so and it's got It's got in here like you can see the the content field up there is like the the baffy our root CID of our our data for our dag um and we call them shards but they're like because you can Shard the bag across multiple car files you can have more than one car file with with a chunk a chunk of the the uh the dag in it um so the shards have a um a a a Sid so that's the Sid of the car or the hash of the car that we were talking about and then they have a list of all of the multi hases that you can find in that car and their offsets and lengths so that that's cool um because we got lengths in the index uh and we also know which uh we should know the the Sid of the car so um so that's good as well um cool so synthetic hashes uh if you if you take the the the like um the the bites of the the um the multi has for the root uh root of the dag and just sort of concatenate it with like some arbitrary string bites uh and then take the hash of that and use that as a multi has you can actually like store um M multiple bits of metadata for any given CID by creating another CID with a with a known uh thing so this is like for for the index we might create a multi has that that has this this in it for the location we might have a a kind of something more like this um and why why might we want to do that um well in in ipni you can uh you can delete uh uh entry delete it's a blockchain so there's no real deleting but you can kind of edit stuff so you might want to uh so for something like the location URL that's fairly IM mutable right like you might like whoever's storing that might end up dropping it we might want to move it somewhere else um and what we'd want to do then is kind of update the the in we'd like to like delete that piece that piece of information from from um IP Andi essentially but we don't want to then um like also delete the information about the index because that's pretty that pretty immutable it is immutable and um uh and so that just gives us some nice um nice separation of concerns right um all right Co final bit thank you for um holding on in so in in stacha we will have many nodes in the in the network um we we have like retrieval nodes object storage nodes um and uh other no like we have Fisher nodes and and all sorts right um just putting what what what we're do what we're using right now here um we've got a client coming along they're asking for a particular that that root hash that we were talking about um uh they ask a retrieval node typically retrieval node um would then ask ipni for like the that special index the EUR like the URL of where it can fetch that thing we'd get back some information about like the the car that that it's in you can see that in this case that hash is in and got put to the object storage number one uh node uh and we've got information in the index about the blocks that are in that um in that car file uh and their by offsets so once the retrieval node has that information it has everything it needs to be able to satisfy the request so it can then go to an object storage node and be like okay get get that car but this bite this range of bites because I want that block and it might make another one for the second block or like it might not like it might make a request for bytes 0 to 110 because that sort of makes sense right but you get the idea so they could be the distrib like the blocks could be distributed around the car fire you got to make a a cool um it will also support like um what is it uh multi-art bite range requests which are also a thing which is good fun um anyway that aside and then once it's once we've got the bites uh we serve back the cat image uh to the client and um that's all I wanted to say so thank you very much for listening and uh and ho [Applause]