Is there any readme/documentation about this work?

[hoax-killer]

Hi I am interested to understand this work, but not able to decide where to start...
Is there any manual or read me for this work?

Thanks!

[edaniszewski]

@arvind2688
I don't think there is any official readme/manual for this work. Since it was created for a class throughout term, we rushed a lot of it and spent less time writing clean documentation than we should have.

If you haven't already checked it out, this implementation was based off of this paper. It should provide you with the background info you would need to understand the design/implementation/reason for much of the work found in the repo.

If you are looking for more specific details around our implementation, e.g. how we did certain things, why we did things certain ways, then it may be worth spending time going through the actual source code since it is for the most part well-commented.

If you have more questions, feel free to ask and I'll try to answer to the best of my ability..

[hoax-killer]

@edaniszewski
Thanks for that quick response. Yes, I have read the GFS paper and that by googling GFS implementation is how I stumbled upon your repo.

I am trying to run your implementation. But I am not able to understand the starting point.
I see a www/ folder, should I just try to run it on a web server, are there any path dependencies to the Python source, etc.

[edaniszewski]

@arvind2688
Its been a while since I've run this, and unfortunately I spend most of my time working on the master server, so my knowledge on the implementation of the other parts is a bit rusty.

The first thing to know about this implementation is that, unlike other simplified GFS implementations (e.g. http://clouddbs.blogspot.com/2010/11/gfs-google-file-system-in-199-lines-of.html ), our implementation was designed as a networked system, that is to say -- I don't believe the functionality exists for it to run completely locally yet. This means that if you want it to run as designed, you'd need access to networked machines. (we used a cluster of raspberry pis + computer lab machines on a local network switch)

Assuming you have this, what you would need to do is clone the repository/copying the repository onto each machine (there is a way to have an automated push from a central machine and MINIMAL versioning so it would re-push when changes were detected, but I didn't write this and there was no documentation on how to use it so I'm kind of lost on that end.. we also had someone work on setting up cron jobs to run specific components, but again, not my area of focus on this project, though this should be easier to figure out, as it would just be editing the crontab.) Before distributing to each machine, edit the hosts.txt file to add the IP addresses of the machines on the network.

Choose one machine to be the master, run master.py on that machine, on the other machines that will act as chunk servers, run chunkserver.py. Heartbeats should run automatically to validate whether the IPs specified in hosts.txt are reachable/active over some port specified in config.py.

To interact with the system, we have three 'clients'. One is a webserver, as you noted, from the www/ directory. Being on the backend, I didn't spend too much time getting familiar with the specifics of how to get that bit working.. I was mostly concerned with getting the API right. A second option is client.py -- this is a CLI client. A while back, I also wrote a thin GUI, newclient.py which may give insight towards how to interact with it, though back then I had no UI experience, so I'm sure its buggy. Both the CLI and GUI utilize API.py which has all the supported filesystem operations that this implementation can handle.

All of this is from memory, so theres a good chance I missed something.