|
Replies:
15
-
Last Post:
Jul 21, 2006 11:41 AM
by: ulfzibis
|
|
|
|
|
|
|
FS extended attributes and FS notifications in Java.
Posted:
Sep 19, 2005 11:03 PM
|
|
|
Hi all,
There exists a class of problems that cannot be given a scalable solution with current J2SE. These include, for example, an enterprise file management solution, and a desktop search engine.
Imagine we have to develop a web-based interface for file uploading, with metadata storage (author, public/private flag, category, keywords), and with concurrent access to the file repository as a regular filesystem, for further categorization, renaming/moving, sorting, FTP access (important!). Due to missing filesystem extended attributes and filesystem notification support, a J2SE solution is going to be extremely inefficient. (I'm going to explain this further if there is an interest.)
Or, look at Beagle (http://www.beaglewiki.org/Main_Page). This desktop search engine, written in C#, makes extensive use of both Linux EAs and inotify.
I'll have to develop a minimal JNI bindings to POSIX EAs and SGI FAM and/or dnotify/inotify. I suppose there can be a community interest of including those features in Mustang (or maybe in Java 7.0). Please give me a tip where to look further, or, if this is not a correct place for making proposals like that, whom and where do I have to contact for collaboration.
Thank you!
Dimitri
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 20, 2005 5:36 AM
in response to: ochkarik
|
|
|
Could not agree more.
To add to this pile, the file operation methods in the java.io.File class should not just return a boolean indicating success or not. Instead, they should throw a descriptive exception on failure instead.
Currently, if I do File.delete() and get back false, I cannot say if the cause of the problem is insufficient rights or a missing parent directory, or a corrupted file system, or another process still having the file open for I/O (this will cause an error on Windows platforms only).
And by the way: If this is going to be redesigned, File should be an interface with a factory method, so that it can be decorated in order to represent some other object with a directory-like structure.
Regards,
Christian
|
|
|
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 20, 2005 6:06 AM
in response to: ochkarik
|
|
|
Support for extended attributes is part of JSR-203 (and was previously part of JSR-51). Unfortunately there are few visible signs of progress on this JSR.
See http://jcp.org/en/jsr/detail?id=203
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 20, 2005 4:02 PM
in response to: ochkarik
|
|
|
Alan Bateman says:
This is a good (or maybe bad) timing. I've just taken over JSR-203 and will be spinning it up very shortly. One of its goals is to define a new file system interface. As you have identified the current API (essentially java.io.File) only provides very limited access to file attributes. This means that applications currently have to resort to native code to do anything with EAs. File change notification is also very important and a number of file systems now provide some sort of notification mechanism. The goal for JSR-203 is to be ready in good time for Dolphin (Java SE 7.0). This might be too late for what you want to do but worth keeping in mind - maybe you can structure things so that it won't be to difficult to migrate in the future.
(http://forums.java.net/jive/thread.jspa?threadID=1468&tstart=0)
In fact, I'll start implementing POSIX EA bindings in a few days or so. Concerning the above, I would like to discuss some design details.
For example, EAs may not be available for several reasons: filesystem without EAs, no support in kernel, missing libattr.so and so on. We need to check it thoroughly and throw an exception. Probably we'll need to load libattr.so via dlopen()/dlsym() (carry out dynamic linking ourselves).
Further, I'd like to inherit from java.io.File and extend it with getAttributes() method returning an java.util.Properties instance that models EAs (read/write). Perhaps there is a need for higher level abstraction, as there exist NTFS streams and HFS forks - objects of the same nature but with different semantics, and the should probably be covered with the API, also. Codepage issues also need more attention, as POSIX EA API returns byte arrays for both EA names and values, and we need to think over converting them to java.lang.Strings properly.
If you say that now it's too early to discuss details of such depth, that's quite okay, I'll just start implementing everything I need for now, sharing my (future) experience and describing common pitfalls.
Sincerely,
Dimitri
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 20, 2005 4:22 PM
in response to: ochkarik
|
|
|
Please start with delete and renames which do have a usefull error handling, some f* family calls (fstat, funlink, ...) any maybe easier access to fsnc/sync.
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 20, 2005 8:34 PM
in response to: ochkarik
|
|
|
I've been frustrated with java.io.File for quite a while. I started working on a project called JGenFile (http://jgenfile.sourceforge.net) but soon discovered that I was in way over my head and it got abandoned. JGenFile was supposed to be a generic file system interface that allowed that could potentially have implementations for the local file system as well as remote file systems and file systems within file systems (e.g. ZIP files).
I can tell you a couple of issues that came to mind while working on it:
- Progress monitoring: good idea in theory, not so easy to implement in practice. This is the reason why everything in JGenFile is an operation instead of just a method call. All operations that could be monitored would implement a ProgressObservable interface which gave access to the percantage an operation had completed. Unfortunately this made JGenFile more difficult to use among other things.
- Accessing arbitrary metadata: would be useful. Windows has this info available, WinFS could potentially make this much more important.
- Security Information: Unix permissions are fairly standard, but accessing Windows permissions and Unix permissions through a common interface without sacrificing all that extra info in Windows is difficult. Then there were ZIP passwords, SSH keys for SCP, etc. Perhaps a standard FileSecurity interface with basic stuff (isReadable(), isWritable()) and then users can cast with an instanceof check if they want more info (e.g. WindowsFilePermissions, UnixFilePermissions).
- Illegal file names. Is it necessary to attempt to create a file to find out if its name is illegal. Or should there be an interface that can pre-emptively check a file name for legality and possibly even return a reason (e.g. name too long, reserved name). Also what character encoding is used for file names in different file systems.
- Copying between different file systems: file name character encoding, file size limits, a file name being legal in one file system and illegal on another. This is only an issue if multiple file system implementations are allowed.
- Links: knowing that a file is a soft-link to another location in the file system would be useful. Should Windows shortcuts be treated similarly? What if we want to access the binary data of a Windows shortcut instead of following its link?
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 21, 2005 1:24 AM
in response to: prunge
|
|
|
Multiple file systems are quite common: CDR and DVD file systems, as well as for Windows NTFS and FAT, plus of course remote file systems. These all have different limitations on file names.
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 21, 2005 3:35 AM
in response to: ochkarik
|
|
|
I'd really like to be able to monitor a set of files/dirs for changes and also detect the availabilty of entire filesystems (USB drives, CD's, network drives). Sure, my C: drive is always there but I'm always plugging in compact flash cards, USB drives, external drives, etc., and I've got no way in Java to monitor these (e.g. to provide background sync, backup, cataloging, etc)
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 21, 2005 9:26 AM
in response to: hopeless
|
|
|
my two cents..
(i) have some kind of class hierarchy allowing pluggable services so that platform specific behaviour can be surfaced in a controlled way - most notably unix file permissions (setting executablness is a must!) and symbolic link creation/deletion/detection
(ii) make high performance an objective - there are various bugs in the bug database against File for doing overzealous 'stat'-ing. Traversing a directory tree quickly can be important!
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5036988
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 22, 2005 8:36 AM
in response to: ochkarik
|
|
|
The limitations of java.io.File are well recognized. JSR 203 will have to define a way to access file attributes in an efficient manner (some applications need bulk access for example). The challenge is that file systems are very varied. One the replies here mentioned file permissions and that is a good example of where things are very different - some file systems support simple concepts of file owner and basic permissions while others support more complex security with detailed permissions and access control lists. Someone else mentioned that many of the methods don't throw exceptions which makes it impossible to report errors. Yet another reply mentioned naming (very important), and symbolic links. These are all good examples of limitations that need to addressed by JSR-203. As I said, this is Dolphin rather than Mustang. One thing that I would really like is collect is as many use-cases and examples from developers that need to access extended file attributes - are people mostly interested in permissions or are there many people with examples (like Beagle as was mentioned) that need to access other extended attributes?
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 28, 2005 8:04 AM
in response to: alanb
|
|
|
It occurs to me that there is already a Java API that could provide access to attributes (in bulk) and also notification. Namely JNDI. All that is needed is a suitable JNDI provider for the file system (and not the trivial one built on top of File). While this would be more complex to use than a purpose built API for files and file systems, perhaps the extra work in learning to use it would pay off when using JNDI for other purposes such as DNS queries. After all File already covers most of the simple use cases.
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 29, 2005 12:00 AM
in response to: mthornton
|
|
|
Correct. This is just another request for duplicating functionality because the OP is too lazy to inform himself about the available options to solve his problems in the intended way.
More feature creep, more crud, more opaque syntax for users to get frustrated with.
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 29, 2005 2:44 AM
in response to: jwenting
|
|
|
Although I think JNDI is a suitable framework, the JRE does not currently include a suitable provider for file systems. This would need to be added and would involve a significant amount of work. There used to be a file system JNDI provider, that was more of a demo and as it was based on File it provided no more information, nor were events available.
So as far as the OP is concerned JNDI is not currently an available option.
One advantage of using JNDI as a basis is that it could be developed without adding anything to the java.* or javax.* name spaces (one might wish to add some convenience definitions of standard attribute names). So perhaps a public project could be created to develop such a provider and it could be used immediately with the current release of Java. Then when complete it could be added to Java while requiring little or no change to applications already using it. Or it could be decided to leave it as an optional extra to be downloaded by those who needed it.
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
Sep 29, 2005 7:23 AM
in response to: alanb
|
|
|
To some extend, it could be provided through the SPI model that is present in NIO (see java.nio.channels.spi). Sun could provide an out of the box default implementation, but folks are free to find alternatives if they so desire. Google has made noise about wanting something like this to be available for some time now (they even made a point about it in the vote for Mustang). I'm pretty sure they'd created their own version that suit their GoogleFS.
A new package like java.nio.attributes.spi (with a java.nio.attributes) would do the trick. Same goes for security or more so permissions. Then the task is for JCP to create a standard minimal set and optional or recommend properties to be available. You could easily prototype a default SPI that uses java.io.File as a POC or a reference point.
My 2c
|
|
|
|
|
|
|
|
|
Re: FS extended attributes and FS notifications in Java.
Posted:
May 21, 2006 5:06 PM
in response to: noahcampbell
|
|
|
Looks like each time I call a method on File, I end up having a stat() system call.
One could imagine a a more efficient bulk interface that returns a disconnected object that encapsulates the File's attributes, permissions, name, physical/absolute path, size and so on.
Currently I get a stat64() when I access the URL then another stat64() when I ask it's size and another for other operations.
|
|
|
|
|
java.net RSS
