Once I had a task to import a number of data from datafeeds (that are in CSV or XML) into MySQL database. See solution in my another post. Here I want to tell about handling large files.
Datafeeds there are 50-200M each, so reading all file at once was not possible due to memory limitations. So we need to split it into portions. Here is the class, that solves the problem of downloading remote file, saving it locally and then reading by portions. Continue reading ‘Working with large files eficiently’ »
