Welcome to the Java Programming Forums


The professional, friendly Java community. 21,500 members and growing!


The Java Programming Forums are a community of Java programmers from all around the World. Our members have a wide range of skills and they all have one thing in common: A passion to learn and code Java. We invite beginner Java programmers right through to Java professionals to post here and share your knowledge. Become a part of the community, help others, expand your knowledge of Java and enjoy talking with like minded people. Registration is quick and best of all free. We look forward to meeting you.


>> REGISTER NOW TO START POSTING


Members have full access to the forums. Advertisements are removed for registered users.

Results 1 to 4 of 4

Thread: Saving Many, Possibly Overlapping Files Asynchronously

  1. #1
    Junior Member
    Join Date
    Oct 2018
    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Saving Many, Possibly Overlapping Files Asynchronously

    I am trying to create an alternative, file-based data storage method where there are many different config files spread throughout folders, and my program periodically writes to some of them to store data. This would be fine with a single thread writing data, but I have multiple threads running asynchronously that could possibly save data to the same file at the same time.

    While the likelihood of the same file being modified by two threads simultaneously is low, I would rather make sure no issues can arise from it ever, so how would I go about ensuring that no two threads work on the same file at the exact same time?

    For example, if two threads want to modify one file at the same time, the first thread will run, then the second will run afterwards, preferably without blocking the other code running on the second thread if the first one takes a long time.

    I am editing these files through my own API, so I pass a key and value pair to my method, and from there my own code will update the value and save the appropriate file.

    Sorry, I have not worked much with asynchronous threads much and I don't know a whole lot about how to efficiently achieve things like this on a large scale without blocking a bunch of threads and causing a big slow down.

  2. #2
    Member
    Join Date
    Sep 2018
    Location
    Virginia
    Posts
    94
    My Mood
    Cool
    Thanks
    0
    Thanked 14 Times in 14 Posts

    Default Re: Saving Many, Possibly Overlapping Files Asynchronously

    One way would be to have a list of these config files in a MAP and then have then tagged as either locked or not locked. Before writing, the process checks the file to see if it is already "owned" by a process. If not, it takes ownership and designates the file as locked. When done it unlocks the file. The process to lock or unlock the file must by synchronized so that no two processes believe they have ownership at the same time. Any thread which can be granted ownership would need to wait until the file is unlocked. How this is done is up to you.

    You should write this as a separate test program before incorporating it into your code. Check out the synchronizedMap method and other synchronizedXXX methods in the Collections class.

    Regards,
    Jim

  3. #3
    Junior Member
    Join Date
    Oct 2018
    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Re: Saving Many, Possibly Overlapping Files Asynchronously

    Quote Originally Posted by jim829 View Post
    One way would be to have a list of these config files in a MAP and then have then tagged as either locked or not locked. Before writing, the process checks the file to see if it is already "owned" by a process. If not, it takes ownership and designates the file as locked. When done it unlocks the file. The process to lock or unlock the file must by synchronized so that no two processes believe they have ownership at the same time. Any thread which can be granted ownership would need to wait until the file is unlocked. How this is done is up to you.

    You should write this as a separate test program before incorporating it into your code. Check out the synchronizedMap method and other synchronizedXXX methods in the Collections class.

    Regards,
    Jim
    Hello,

    Thank you for your response! I was considering something like locking the files individually, but I was wondering, wouldn't this end up blocking threads if the file writing process took too long? I wasn't sure how to go about making sure I don't end up locking up the program because a file took too long to write to or something.

    I just thought of maybe using a LinkedBlockingQueue that stores an object which contains the file to write to, as well as what to write and using a consumer thread to go through and save the files, however I ended up again at the problem of if a file takes too long, then the consumer thread locks up and nothing saves in time.

    Because of this, I was thinking, perhaps I can have several consumer threads waiting to save any files as well as a ConcurrentHashMap or something that stores file locks for the consumers to check on when taking a new job from the LinkedBlockingQueue.

    Would something like this be advisable, or am I just overthinking this way too much and missing a much simpler solution?

  4. #4
    Member
    Join Date
    Sep 2018
    Location
    Virginia
    Posts
    94
    My Mood
    Cool
    Thanks
    0
    Thanked 14 Times in 14 Posts

    Default Re: Saving Many, Possibly Overlapping Files Asynchronously

    Thank you for your response! I was considering something like locking the files individually, but I was wondering, wouldn't this end up blocking threads if the file writing process took too long? I wasn't sure how to go about making sure I don't end up locking up the program because a file took too long to write to or something.
    If a file takes too long to write you want it to block because you don't want another thread to start writing to the same file. Within the thread you could have a timer and fire off an event for the thread to exit after a certain amount of time. Check out the java.util.timer class.


    I just thought of maybe using a LinkedBlockingQueue that stores an object which contains the file to write to, as well as what to write and using a consumer thread to go through and save the files, however I ended up again at the problem of if a file takes too long, then the consumer thread locks up and nothing saves in time.
    I am not familiar with this as I have never used it. But the timer solution still applies.

    Overall you could have a thread pool of available threads to run to write to the files. If you run out of threads you can simply have a queue for the requests for the next free thread to process. But there is always the potential of either running out of threads and having the writes take too long or having a timer and not completely writing the information. How you deal with that is up to you but if the thread locks up and never releases due to a long write, there is definitely a bug in your code. All writes must eventually end, even with an I/O error.

    Regards,
    Jim

Similar Threads

  1. Can this possibly done?
    By cooljava in forum File I/O & Other I/O Streams
    Replies: 3
    Last Post: August 25th, 2014, 10:01 AM
  2. Reason for swing component overlapping?
    By Furious5k in forum AWT / Java Swing
    Replies: 1
    Last Post: November 6th, 2011, 08:40 AM
  3. How would i remove overlapping lists
    By redzero36 in forum Java Theory & Questions
    Replies: 19
    Last Post: July 21st, 2011, 04:42 PM
  4. Overlapping multiple periods
    By Atei in forum Algorithms & Recursion
    Replies: 1
    Last Post: May 3rd, 2011, 12:35 PM
  5. Overlapping windows? in a Jframe?
    By chronoz13 in forum AWT / Java Swing
    Replies: 6
    Last Post: November 21st, 2009, 12:01 PM