Welcome to the Java Programming Forums


The professional, friendly Java community. 21,500 members and growing!


The Java Programming Forums are a community of Java programmers from all around the World. Our members have a wide range of skills and they all have one thing in common: A passion to learn and code Java. We invite beginner Java programmers right through to Java professionals to post here and share your knowledge. Become a part of the community, help others, expand your knowledge of Java and enjoy talking with like minded people. Registration is quick and best of all free. We look forward to meeting you.


>> REGISTER NOW TO START POSTING


Members have full access to the forums. Advertisements are removed for registered users.

Results 1 to 2 of 2

Thread: How to improve FTP upload speed for multiple files.

  1. #1
    Junior Member
    Join Date
    Jul 2013
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default How to improve FTP upload speed for multiple files.

    I implemented java code to upload files to server with org.apache.commons.net.ftp.FTPClient (commons-net-3.3.jar).

    Following is example code:

    // create instance of FTPClient
    FTPClient ftp = new FTPClient();
     
    ftp.setControlEncoding("UTF-8");
    ftp.setDefaultTimeout(30000);
     
     
    // connect to server
    try
    {
        ftp.connect("10.1.1.1", 21);
    }
    catch(Exception e)
    {
        System.out.println("Cannot connect to server");
        return;
    }
     
    // login to server
    if (!ftp.login("username", "password"))
        {
            ftp.logout();
            System.out.println("Cannot login to server");
            return;
        }
     
    try
    {
    	ftp.setFileType(FTP.BINARY_FILE_TYPE, FTP.BINARY_FILE_TYPE);
    	ftp.setFileTransferMode(FTP.BINARY_FILE_TYPE);
    	ftp.enterLocalPassiveMode();
    }
    catch(Exception e)
    {
    }
    // create directory on server
    // dirs is list of required directories on server
    for (String dir : dirs)
        {
            try
            {
                ftp.makeDirectory(dir);
            }
            catch(IOException e)
            {
            }
        }
     
     
    // files is a map of local file and string of remote file
    // such as 
    // file on client is "C://test/a.txt"
    // location on server is "/test/a.txt"
    for (Map.Entry<File, String> entry : files.entrySet())
        {
            File localFile = entry.getKey();
            String remoteFile = entry.getValue();
     
            InputStream input = null;
            try
            {
                input= new FileInputStream(localFile);
                ftp.storeFile(remoteFile, input);
            }
            catch (Exception e)
            {
                try
                {
                    ftp.deleteFile(remoteFile);
                }
                catch (IOException e1)
                {
                }
            }
            finally
            {
                if (input != null)
                {
                    try
                    {
                        input.close();
                    }
                    catch (IOException e)
                    {
                    }
                }
            }
        }
     
    // disconnect
    if (ftp != null && ftp.isConnected())
        {
            try
            {
                ftp.disconnect();
            }
            catch (IOException f)
            {
                // do nothing
            }
        }

    I try to upload 100 files (each file is about 2-10 KB). It took about 174375 - 210203 millisec. I would like to improve the speed.

    How should I do?

    - Change library? What is the powerful FTP client class library for uploading multiple files (only open source or free library)?

    - Use multiple threads? How can I implement ftp upload function with multiple threads? Could someone show me an example? I am a new for multiple threading programming.

    I try to use this code for multiple threads:
    final CountDownLatch latch = new CountDownLatch(files.size());
    ExecutorService pool = Executors.newFixedThreadPool(5);
    final org.apache.commons.net.ftp.FTPClient client = ftp;
    for (Map.Entry<File, String> entry : files.entrySet())
    {
    	final File localFile = entry.getKey();
    	final String remoteFile = entry.getValue();
     
    	pool.execute(new Runnable() {
    		public void run()
    		{
    			try
    			{
    				uploadFile(client, localFile, remoteFile);
    			}
    			catch (Exception e)
    			{
     
    			}
    			finally
    			{
    				latch.countDown();
    			}
    		}
    	});
    }
    try
    {
    	// waiting for all threads finish
    	latch.await();
    }
    catch (Exception e)
    {
     
    }

    uploadFile method:
    private void uploadFile(org.apache.commons.net.ftp.FTPClient ftp, File payLoad, String remoteLoc)
    {
    	InputStream input = null;
    	try
    	{
    		input = new FileInputStream(payLoad);
    		ftp.storeFile(remoteLoc, input);
    	}
    	catch (Exception e)
    	{
    		try
    		{
    			ftp.deleteFile(remoteLoc);
    		}
    		catch (Exception e1)
    		{
    		}
    		e.printStackTrace();
    	}
    	finally
    	{
     
    		if (input != null)
    			try
    			{
    				input.close();
    			}
    			catch (Exception e)
    			{
    			}
    	}
    }

    Is it correct?

    Can anyone help me?

    Thanks in advance
    Darknight


  2. #2
    Super Moderator helloworld922's Avatar
    Join Date
    Jun 2009
    Posts
    2,896
    Thanks
    23
    Thanked 619 Times in 561 Posts
    Blog Entries
    18

    Default Re: How to improve FTP upload speed for multiple files.

    Using multiple threads will likely not help since almost always the network is the limiting factor by several orders of magnitude (you only get one network, regardless of how many threads you have).

    I've never used apache commons ftp client, or looked into the ftp protocol in detail, but in general:

    1. Try to coalesce your commands. Bulk network operations are usually faster than non-bulk operations.
    2. See if there's some sort of compression support in ftp. The less data you have to transfer, the better, and as stated above CPU time almost never is the bottleneck.
    3. Perhaps your network really is just slow. Unfortunately, the only solution is to get a faster network (which usually means you're stuck with the poor performance).

Similar Threads

  1. Replies: 0
    Last Post: March 27th, 2012, 08:00 AM
  2. Scanning large FTP directories with apache.commons.net.ftp
    By daniel_el in forum Java Theory & Questions
    Replies: 3
    Last Post: February 10th, 2012, 06:40 AM
  3. HOW TO UPLOAD JAVA FILES IN NETBEANS
    By ordonezc78 in forum Java IDEs
    Replies: 2
    Last Post: March 19th, 2011, 09:05 AM
  4. simple ftp server and ftp client
    By simontkk2005 in forum Java Networking
    Replies: 4
    Last Post: January 26th, 2011, 10:29 AM