Thread: Saving Many, Possibly Overlapping Files Asynchronously

I am trying to create an alternative, file-based data storage method where there are many different config files spread throughout folders, and my program periodically writes to some of them to store data. This would be fine with a single thread writing data, but I have multiple threads running asynchronously that could possibly save data to the same file at the same time.

While the likelihood of the same file being modified by two threads simultaneously is low, I would rather make sure no issues can arise from it ever, so how would I go about ensuring that no two threads work on the same file at the exact same time?

For example, if two threads want to modify one file at the same time, the first thread will run, then the second will run afterwards, preferably without blocking the other code running on the second thread if the first one takes a long time.

I am editing these files through my own API, so I pass a key and value pair to my method, and from there my own code will update the value and save the appropriate file.

Sorry, I have not worked much with asynchronous threads much and I don't know a whole lot about how to efficiently achieve things like this on a large scale without blocking a bunch of threads and causing a big slow down.

Originally Posted by jim829

One way would be to have a list of these config files in a MAP and then have then tagged as either locked or not locked. Before writing, the process checks the file to see if it is already "owned" by a process. If not, it takes ownership and designates the file as locked. When done it unlocks the file. The process to lock or unlock the file must by synchronized so that no two processes believe they have ownership at the same time. Any thread which can be granted ownership would need to wait until the file is unlocked. How this is done is up to you.

You should write this as a separate test program before incorporating it into your code. Check out the synchronizedMap method and other synchronizedXXX methods in the Collections class.

Regards,
Jim

Hello,

Thank you for your response! I was considering something like locking the files individually, but I was wondering, wouldn't this end up blocking threads if the file writing process took too long? I wasn't sure how to go about making sure I don't end up locking up the program because a file took too long to write to or something.

I just thought of maybe using a LinkedBlockingQueue that stores an object which contains the file to write to, as well as what to write and using a consumer thread to go through and save the files, however I ended up again at the problem of if a file takes too long, then the consumer thread locks up and nothing saves in time.

Because of this, I was thinking, perhaps I can have several consumer threads waiting to save any files as well as a ConcurrentHashMap or something that stores file locks for the consumers to check on when taking a new job from the LinkedBlockingQueue.

Would something like this be advisable, or am I just overthinking this way too much and missing a much simpler solution?