Welcome to the Java Programming Forums


The professional, friendly Java community. 21,500 members and growing!


The Java Programming Forums are a community of Java programmers from all around the World. Our members have a wide range of skills and they all have one thing in common: A passion to learn and code Java. We invite beginner Java programmers right through to Java professionals to post here and share your knowledge. Become a part of the community, help others, expand your knowledge of Java and enjoy talking with like minded people. Registration is quick and best of all free. We look forward to meeting you.


>> REGISTER NOW TO START POSTING


Members have full access to the forums. Advertisements are removed for registered users.

Results 1 to 17 of 17

Thread: Splitting large video files into small chunks and combining them

  1. #1
    Junior Member
    Join Date
    Nov 2013
    Posts
    7
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Default Splitting large video files into small chunks and combining them

    Hi All,

    Thank you for viewing my thread. I have searched for hours but cannot find any information on this topic.

    I need to upload very large video files. The files can be something like 10GB in size, currently .mpeg format but they may be .mp4 format later on too.

    Since the files will be uploaded remotely using using internet from the clients mobile device (3G or 4G in UK) I need to split the files into smaller chunks. This way if there is a connection issue (very likely as the files are large) I can resume the upload without doing the whole 10gb again.

    My question is, what can I use to split a large file like 10Gb into smaller chunks like 200mb, uplaod the files and then combine them again? Is there a jar file that I can use. i cant install stuff like ffmpeg on the clients laptops.

    Thank you for your time.

    Kind regards,
    Imran


  2. #2
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    25,042
    Thanks
    63
    Thanked 2,708 Times in 2,658 Posts

    Default Re: Splitting large video files into small chunks and combining them

    You could write a utility program to split the large file into smaller chunks to be sent and then to join the chunks back together into the single file.

    (I have a project like that in mind for copying large files from a PC to an Android tablet because of a a copy utility's 30M limit on file size. I haven't started on it yet)
    If you don't understand my answer, don't ignore it, ask a question.

  3. #3
    Junior Member
    Join Date
    Nov 2013
    Posts
    7
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Default Re: Splitting large video files into small chunks and combining them

    Hi Norm,

    Thank you for replying to my thread. Can you point in the right direction on what Java classes/api I can use to split and merge video files please? I ahve looked at Apache Commons but cannot find if its possible.

    Kind regards,
    Imran

  4. #4
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    25,042
    Thanks
    63
    Thanked 2,708 Times in 2,658 Posts

    Default Re: Splitting large video files into small chunks and combining them

    My suggestion has nothing to do with the contents of the file. It is strictly byte by byte.
    Copy the first n bytes to a file
    copy the next n bytes to another file
    etc

    Then read the short files one by one and append the bytes to the new large file.
    If you don't understand my answer, don't ignore it, ask a question.

  5. The Following User Says Thank You to Norm For This Useful Post:

    imran.ali@adamm.eu (November 25th, 2013)

  6. #5
    Forum VIP
    Join Date
    Jul 2010
    Posts
    1,676
    Thanks
    25
    Thanked 329 Times in 305 Posts

    Default Re: Splitting large video files into small chunks and combining them

    When you remember that all files are just long lists of bytes (including media files), anything is possible, lol.
    Don't get caught up on the idea of a file having some sort of extension which makes it a video file, or whatever type of file. Regardless of the extension or type of file, they are all just bytes.
    A perfect example of this basic idea: get a small mp4 file or something, remove its extension (so it is a file with no extension), send it to a friend or something, get them to put the mp4 extension back on, and the video file will operate perfectly, as if you never changed it's extension in the first place. The extension is really just there to tell the OS how to use a given file, it means nothing for your purposes (although, when you recreate the file, you will need to use the same extension, because different video formats are encoded and/or compressed in different ways, and that extension tells the system how the video file is encoded and compressed).
    NOTE TO NEW PEOPLE LOOKING FOR HELP ON FORUM:

    When asking for help, please follow these guidelines to receive better and more prompt help:
    1. Put your code in Java Tags. To do this, put [highlight=java] before your code and [/highlight] after your code.
    2. Give full details of errors and provide us with as much information about the situation as possible.
    3. Give us an example of what the output should look like when done correctly.

    Join the Airline Management Simulation Game to manage your own airline against other users in a virtual recreation of the United States Airline Industry. For more details, visit: http://airlinegame.orgfree.com/

  7. The Following User Says Thank You to aussiemcgr For This Useful Post:

    imran.ali@adamm.eu (November 25th, 2013)

  8. #6
    Junior Member
    Join Date
    Nov 2013
    Posts
    7
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Default Re: Splitting large video files into small chunks and combining them

    Hi Norm and Aussiemcgr,

    Thank you for getting back. Its much more clearer to me know. I will convert the video file into a byte array and then convert it back to the video. I will give this a try and report back in a few days to let you know how I get on.

    Kind regards,
    Imran

  9. #7
    Junior Member
    Join Date
    Nov 2013
    Posts
    7
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Default Re: Splitting large video files into small chunks and combining them

    Hi,

    Ive been reading up on socket programming in Java. What do you think about sending multiple byte[] array's over a socket compared with sending byte files over ftp?

    I need a way to resume the upload if there is a connection issue.
    With the byte files approach I can do a checksum of the files on both sides to make sure its moved over without issues. I can also keep track on how many files are moved over in the db so I can resume if there was a issue. The problem with this approach is that its going to create loads and loads of small files for a 10gb file.

    Im not sure how I would keep track using sockets? Do you think this may be a better approach?

    I basically trying to replicate a dropbox like feature.

    Thank you for your time,
    Imran

  10. #8
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    25,042
    Thanks
    63
    Thanked 2,708 Times in 2,658 Posts

    Default Re: Splitting large video files into small chunks and combining them

    how I would keep track using sockets?
    Define a custom protocol for sending the blocks. Add a header that has a block number and size before the bytes being sent.
    If you don't understand my answer, don't ignore it, ask a question.

  11. #9
    Forum VIP
    Join Date
    Jul 2010
    Posts
    1,676
    Thanks
    25
    Thanked 329 Times in 305 Posts

    Default Re: Splitting large video files into small chunks and combining them

    With the byte files approach I can do a checksum of the files on both sides to make sure its moved over without issues. I can also keep track on how many files are moved over in the db so I can resume if there was a issue. The problem with this approach is that its going to create loads and loads of small files for a 10gb file.
    As a final check, yes. But while uploading, you will probably find that exceptions are thrown if there are any issues (perhaps connection exceptions or something). Catch these exceptions, and do something with them to indicate to you that a problem occurred. You will have to explore all the different ways your scenarios can fail and account for them.

    Combining the proper exception handling with Norm's custom protocol suggestion should weed out most of the potential issues.
    NOTE TO NEW PEOPLE LOOKING FOR HELP ON FORUM:

    When asking for help, please follow these guidelines to receive better and more prompt help:
    1. Put your code in Java Tags. To do this, put [highlight=java] before your code and [/highlight] after your code.
    2. Give full details of errors and provide us with as much information about the situation as possible.
    3. Give us an example of what the output should look like when done correctly.

    Join the Airline Management Simulation Game to manage your own airline against other users in a virtual recreation of the United States Airline Industry. For more details, visit: http://airlinegame.orgfree.com/

  12. #10
    Junior Member
    Join Date
    Nov 2013
    Posts
    7
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Default Re: Splitting large video files into small chunks and combining them

    Thanks for getting back.

    Im trying to make it easier for myself and use FTP4J which supports restart. But I cannot get it to work. It works fine when uploading the whole file. But when I stop it and provide it a offset (number of bytes that was transferred) it just hangs. No errors.

    Here is my code:

     
    import it.sauronsoftware.ftp4j.FTPClient;
    import java.io.File;
     
    public class FTP4J {
     
    	private static String server = ""; 
    	private static String username = ""; 
    	private static String password = "";
     
    	private String local_dir = "";
    	private String remote_dir = ""; 
     
        public static void main(String[] args) {    	
        	String filename = "small.mp4";
        	boolean append = true;
        	(new FTP4J()).uploadFile(filename, append);
        }
     
        public void uploadFile(String filename, boolean appendIt) {    	
            FTPClient client = null;
            try {
            	client = new FTPClient();
            	client.connect(server);
            	client.login(username, password);
     
            	//if(client.isResumeSupported()) {
            		//System.out.println("Resume is supported");
            	//}
     
                 if (client != null) {
                    try {
                        client.setPassive(true);	
                        client.setType(FTPClient.TYPE_BINARY);
                        client.changeDirectory(remote_dir);
     
                        String filePath = local_dir+filename;                      
                        File fileUpload = new File(filePath);
     
                        //Upload the file
                        if(appendIt) {
                        	System.out.println("Appending the " + filePath + " to Remote Machine");
                        	long offset = 3803616; // taken from server
                        	client.upload(fileUpload, offset, new MyFTPListener());
                        } else {
                        	System.out.println("Uploading the " + filePath + " to Remote Machine");
                        	client.upload(fileUpload, new MyFTPListener());
                        }
     
                        System.out.println("Successfully Uploaded the " + filePath + " File to Remote Machine");
                    } catch (Exception e) {
                            System.err.println("ERROR : Error in Transfering File to Remote Machine");
                            System.exit(3);
                    }
                }
            } catch (Exception e) {
                System.err.println("ERROR : Error in Connecting to Remote Machine");
                System.exit(1);
            }
     
            finally {
                if (client != null) {
                    try {
                        client.disconnect(true);
                    }
                    catch (Exception e) {
                        System.err.println("ERROR : Error in disconnecting the Remote Machine");
                    }
                }
            }
            System.exit(0);
        }

    Please help me find the issue. Many thanks

    Kind regards,
    Imran

  13. #11
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    25,042
    Thanks
    63
    Thanked 2,708 Times in 2,658 Posts

    Default Re: Splitting large video files into small chunks and combining them

    I'd go with the breaking the huge file into parts, sending the parts and then putting them back together, instead of trying to send Gigabytes.

    Otherwise contact the authors of the code you are trying to use.
    If you don't understand my answer, don't ignore it, ask a question.

  14. #12
    Forum VIP
    Join Date
    Jul 2010
    Posts
    1,676
    Thanks
    25
    Thanked 329 Times in 305 Posts

    Default Re: Splitting large video files into small chunks and combining them

    Hanging with no exception could indicate an infinite loop or some sort of waiting.

    Are you able to provide us with the console output? You have print statements in your code. That console output could tell us where the code hangs.
    NOTE TO NEW PEOPLE LOOKING FOR HELP ON FORUM:

    When asking for help, please follow these guidelines to receive better and more prompt help:
    1. Put your code in Java Tags. To do this, put [highlight=java] before your code and [/highlight] after your code.
    2. Give full details of errors and provide us with as much information about the situation as possible.
    3. Give us an example of what the output should look like when done correctly.

    Join the Airline Management Simulation Game to manage your own airline against other users in a virtual recreation of the United States Airline Industry. For more details, visit: http://airlinegame.orgfree.com/

  15. #13
    Junior Member
    Join Date
    Nov 2013
    Posts
    7
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Default Re: Splitting large video files into small chunks and combining them

    Thanks for getting back.

    This is what the print statements print. After line 3 it hands for like 15 minutes. I have looked into breaking the big files but the issue I have with that is with a 10gb file it will create 100000 files, 100mb each. Thats a lot of files. I guess I can create one file at a time move it over.

    1. Resume is supported
    2. Appending the /Users/adamm001/Documents/Projects/ADaMM CCTV Uploader/upload/small.mp4 to Remote Machine
    3. TRANSFER-STATUS: File transfer started...
    4. TRANSFER-STATUS: File transfer failed...
    5. ERROR : Error in Transfering File to Remote Machine

    I have created the following listener class which the above code uses:
    package controller;
     
    import it.sauronsoftware.ftp4j.FTPDataTransferListener;
     
    public class MyFTPListener implements FTPDataTransferListener
    {
     
            public void started()
            {
                    // Transfer started
                    System.out.println("TRANSFER-STATUS: File transfer started...");
            }
     
            public void transferred(int length)
            {
                    // Yet other length bytes has been transferred since the last time this
                    // method was called
            }
     
            public void completed()
            {
                    // Transfer completed
                    System.out.println("TRANSFER-STATUS: File transfer completed...");
            }
     
            public void aborted()
            {
                    // Transfer aborted
                    System.err.println("TRANSFER-STATUS: File transfer aborted...");
            }
     
            public void failed()
            {
                    // Transfer failed
                    System.err.println("TRANSFER-STATUS: File transfer failed...");
            }
    }


    Kind regards,
    Imran

  16. #14
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    25,042
    Thanks
    63
    Thanked 2,708 Times in 2,658 Posts

    Default Re: Splitting large video files into small chunks and combining them

    ERROR : Error in Transfering File to Remote Machine
    You need to add a call to the printStackTrace() method in the catch block to get the full text of the error message.
    If you don't understand my answer, don't ignore it, ask a question.

  17. #15
    Forum VIP
    Join Date
    Jul 2010
    Posts
    1,676
    Thanks
    25
    Thanked 329 Times in 305 Posts

    Default Re: Splitting large video files into small chunks and combining them

    I guess I can create one file at a time move it over.
    Create a file, move it over, delete the file, rinse and repeat.
    100mb files are still pretty large, at least for internet transfer they are.
    NOTE TO NEW PEOPLE LOOKING FOR HELP ON FORUM:

    When asking for help, please follow these guidelines to receive better and more prompt help:
    1. Put your code in Java Tags. To do this, put [highlight=java] before your code and [/highlight] after your code.
    2. Give full details of errors and provide us with as much information about the situation as possible.
    3. Give us an example of what the output should look like when done correctly.

    Join the Airline Management Simulation Game to manage your own airline against other users in a virtual recreation of the United States Airline Industry. For more details, visit: http://airlinegame.orgfree.com/

  18. #16
    Junior Member
    Join Date
    Nov 2013
    Posts
    7
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Default Re: Splitting large video files into small chunks and combining them

    This is what I get:

    Resume is supported
    Appending the /Users/adamm001/Documents/Projects/ADaMM CCTV Uploader/upload/small.mp4 to Remote Machine
    TRANSFER-STATUS: File transfer started...
    TRANSFER-STATUS: File transfer failed...
    it.sauronsoftware.ftp4j.FTPException [code=451, message= small.mp4: Append/Restart not permitted, try again]
    at it.sauronsoftware.ftp4j.FTPClient.upload(FTPClient .java:2794)
    at it.sauronsoftware.ftp4j.FTPClient.upload(FTPClient .java:2586)
    at controller.FTP4J.uploadFile(FTP4J.java:68)
    at controller.FTP4J.main(FTP4J.java:19)

  19. #17
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    25,042
    Thanks
    63
    Thanked 2,708 Times in 2,658 Posts

    Default Re: Splitting large video files into small chunks and combining them

    it.sauronsoftware.ftp4j.FTPException [code=451, message= small.mp4: Append/Restart not permitted, try again]
    You should contact the authors of the software about that.
    If you don't understand my answer, don't ignore it, ask a question.

Similar Threads

  1. how to compare the large text files ?
    By vamshi in forum Java Theory & Questions
    Replies: 4
    Last Post: July 14th, 2013, 06:13 AM
  2. Replies: 0
    Last Post: November 16th, 2012, 05:25 AM
  3. How to use MappedByteBuffer efficiently with extremely large files
    By Vyse in forum File I/O & Other I/O Streams
    Replies: 3
    Last Post: September 13th, 2012, 08:44 PM
  4. Replies: 2
    Last Post: March 16th, 2011, 06:32 AM
  5. Fitting a large primitive into a small reference variable
    By Phobia in forum Java Theory & Questions
    Replies: 15
    Last Post: October 23rd, 2009, 03:10 PM

Tags for this Thread