glent , 2 hours ago

@nxskok @joncounts @ml @academicchatter @plantscience I think editting CSV is a poor idea, rather than using it as an interchange and archive format where no common format exists.

For editing I'd suggest a trivial program in Python or R. Because then there is a record of the intention of edits made to the collected data (doubly so if a Jupyter notebook is used to host the program). Using a program also makes it possible to have a CSV of removed data, with the program and version which first removed that data. All this assists the repeatability of the analysis.

For eyeballing unit data I'd suggest a small program to read CSV and save YAML, as that tagged vertical format is a lot easier to read than a horizontal format. It's also a useful format to quote in an edit program as the example of why the edit exists.

One of the annoyances of CSV is the lack of modern data types. So as an interchange format it really needs the file and a sample program which reads it and assigns data types to the columns.

glent , 2 hours ago

@ml @academicchatter @plantscience When using Python for analysis, create a module for the analysis system. This can do drudge work like read a dataset and apply types to it, or hold this analysis' defaults for preparing the graphing system (eg, the printed format for each of a category's names).

Eg:

import traffic2024
df = traffic2024.df()

now ready to plot the data in df

can hide a lot of what might usually be cut-and-pasted. Cut-and-pasting code can make fixing errors inconsistent, and makes it hard to improve the analysis.

You can do this in Jupyter notebooks too, which makes those notebooks come immediately to the point about the analysis to be done.

glent , 2 months ago

@Alexlee @academicchatter Punctures a hole in the AI claim that copyright infringement isn't happening, when the entire AI response is based on a popular fictional article by Luke Burns.