File preprocessing/model execution does not work as expected

[j4freeman]

Awesome project, really exciting and innovative work! In the current master version the project cannot correctly parse new input files and will also fail on the default new york dataset due to indexing issues.

Went ahead and wrote (what I'm pretty sure are) the needed fixes in a fork, happy to raise a pull request if needed. Diffs from current master are here: https://github.com/j4freeman/DeepGravity/pull/1/files

[MassimilianoLuca]

Dear j4freeman,
Thanks for the feedback! That is strange. We tested the default new york data multiple times and it worked in our case. Anyhow, we recently pushed a new version of the code in the master and we have no issues in running this code with the default data. Please let us know if the problem persists!

[j4freeman]

Think we're referring to different things - agree the default NY data works fine, but I'm referring to generating new data with provided input files. Per the documentation, if I provide the relevant output areas/flow/features files for a new area, I would expect DeepGravity to process that and write the pkl/etc results to the data/.../processed directory which could be read in to the model. This currently doesn't happen as lines 136 and 137 in utils.py are commented out, and even when uncommented _compute_support_files does not work as expected.

For example, if I provide a directory data/kansas containing features.csv, flows.csv, and output_areas.geojson, and execute:
python main.py --dataset kansas --oa-id-column GEOID --flow-origin-column geoid_o --flow-destination-column geoid_d --flow-flows-column pop_flows --epochs 10 --device cpu --mode train --tessellation-size 10000 --tessellation-area kansas

The process will fail with:
No such file or directory: './data/kansas/processed/tileid2oa2handmade_features.json'

As the support files were never computed.

[ajahanba]

Thank you for your code, I am able to run it on the New York data as it has the processed folder, however running your code on any other city is impossible due to the code dependency to the processed folder content with no description of what those are. In the main page you mention that the processed folder is created by the code (on the first run). this is not true.

[mrabhinavagarwal]

@j4freeman for me the problem was resolved when I created a processed folder under Kansas directory. Can you try the same?