Welcome to the Java Programming Forums


The professional, friendly Java community. 21,500 members and growing!


The Java Programming Forums are a community of Java programmers from all around the World. Our members have a wide range of skills and they all have one thing in common: A passion to learn and code Java. We invite beginner Java programmers right through to Java professionals to post here and share your knowledge. Become a part of the community, help others, expand your knowledge of Java and enjoy talking with like minded people. Registration is quick and best of all free. We look forward to meeting you.


>> REGISTER NOW TO START POSTING


Members have full access to the forums. Advertisements are removed for registered users.

Results 1 to 4 of 4

Thread: Scanning large FTP directories with apache.commons.net.ftp

  1. #1
    Junior Member
    Join Date
    Feb 2012
    Posts
    4
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default Scanning large FTP directories with apache.commons.net.ftp

    Hi,

    I'm using apache.commons.net.ftp.FTPClient to scan an FTP directory which can contain thousands of files which have been written over several months.

    I'll only ever be interested in a recent subset of those files though (about a week's worth at most), but I'm having to loop through thousands of instances which is taking unacceptably long (over an hour). Is there any way that I can eliminate the older files on mass without having to painfully iterate through each one? Here's the code I'm currently using to do the date checks...

    client.cwd(directory);
    ftpFiles = client.listFiles(directory);

    for (FTPFile ftpFile : ftpFiles) {
    String name = ftpFile.getName();
    if (ftpFile.getType() == FTPFile.FILE_TYPE) {
    Date today = new Date();
    long dateDiffHours = (today.getTime() - ftpFile.getTimestamp().getTimeInMillis())/(1000 * 60 * 60);
    if (dateDiffHours>(argsQueryHours+2)){}else{
    //do something with the file because it's recent enough

    Any help would be really appreciated! Thanks,

    Daniel


  2. #2
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    25,042
    Thanks
    63
    Thanked 2,708 Times in 2,658 Posts

    Default Re: Scanning large FTP directories with apache.commons.net.ftp

    Is there any way that I can eliminate the older files on mass without having to painfully iterate through each one
    If you have a list of the filenames, how do you determine which file you want to process and which file you want to skip?
    What is in the ftpFiles object? Is there something local in the object that can be used to select a file?
    I assume that the problem is with the getTimeStamp() method requires a turnaround with the server.
    Is there a way to get a directory list from the server that will have the names and file's date all in one file obtained in one transaction?

    My simple FTP tool returns a dir list using the LISt command:

    drwxr-x---    2 a3180082   a3180082         4096 Feb  5 07:42 .
    drwx--x--x    3 a3180082   a3180082         4096 Dec  6 16:38 ..
    -rw-r--r--    1 a3180082   a3180082           91 Dec  6 16:04 .htaccess
    -rw-r--r--    1 a3180082   a3180082          363 Dec  7 13:38 AppletReader.html
    -rw-r--r--    1 a3180082   a3180082           41 Feb  5 07:42 AppletTestURL.txt
    -rw-r--r--    1 a3180082   a3180082        24016 Dec  6 16:39 ImagesInBlob.jar
    -rw-r--r--    1 a3180082   a3180082       324867 Dec  6 16:39 SmallBlob.dat
    -rw-r--r--    1 a3180082   a3180082          297 Dec  6 16:39 SmallBlob.html
    -rw-r--r--    1 a3180082   a3180082         8062 Dec  6 16:04 default.php
    -rw-r--r--    1 a3180082   a3180082         3676 Dec  7 13:38 sAppletReader.jar

    What is a "FTP directory"? Is that a directory on a FTP server?

  3. #3
    Junior Member
    Join Date
    Feb 2012
    Posts
    4
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default Re: Scanning large FTP directories with apache.commons.net.ftp

    Hi Norm,

    Thanks for the response.

    ftpFiles is a list of FTPFile objects as specified here: FTPFile (Commons Net 3.0.1 API)

    Unfortunately I can't find anything that returns names and modified dates in one method though. So I'm having to iterate through each file in ftpFiles and examine the date of each.

    Can't find a LIST command that doesn't involve me having to loop through a list of files afterwards

    Any ideas?

  4. #4
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    25,042
    Thanks
    63
    Thanked 2,708 Times in 2,658 Posts

    Default Re: Scanning large FTP directories with apache.commons.net.ftp

    The LIST command returned the file that I posted. Does the API have a way to use FTP commands directly?

    org.apache.commons.net.ftp
    Class FTPCommand

    Has the LIST command in its list of commands.

Similar Threads

  1. need help with 'org.apache.commons.net.ftp.FTPClient'
    By rtumatt in forum File I/O & Other I/O Streams
    Replies: 1
    Last Post: May 22nd, 2013, 07:02 PM
  2. Uploading File to server ( Apache.commons.net)
    By quirell in forum Java Networking
    Replies: 5
    Last Post: November 11th, 2012, 11:06 PM
  3. Merging PDF Files using PDFBox in Sub/Directories
    By blivori in forum What's Wrong With My Code?
    Replies: 10
    Last Post: July 26th, 2011, 11:09 AM
  4. Scanning Document Issue
    By redvenice in forum What's Wrong With My Code?
    Replies: 2
    Last Post: August 20th, 2010, 08:18 AM
  5. How to show empty directories in JTree ?
    By ni4ni in forum AWT / Java Swing
    Replies: 1
    Last Post: April 30th, 2010, 12:55 AM