How do you handle errors at resource release? When you close a file, the final writes take place, and they can fail. What's the idiom in Rust for getting them out?
Python's "with" clause, and the way it interacts with exceptions, is the only system I've seen that gets this right for the nested case.
> How do you handle errors at resource release? When you close a file, the final writes take place, and they can fail. What's the idiom in Rust for getting them out?
That is unclear. Currently, `File::drop` ignores all errors and drops them on the floor ([unix], [windows]). This is a concern both long-standing and ongoing[0].
The fact that this is not resolved after over a year is concerning to me. At some point you have to make a decision and implement a solution, even if not everyone agrees 100% on which solution to chose.
Letting this slide for this long is a very bad sign. I’ve been a big Rust fan for my hobby projects, but the whole point of Rust is effortless correctness and safety. The more I encounter bugs and issues that have no near term solution planned, the more confidence I must admit I’m losing in their bug vs feature work prioritization scheme.
For example, it seems sometimes that Rust management would rather focus on cool new language enhancements / rewrite projects, than fix major bugs (sometimes even major borrow checker bugs, or random segfaults created in correct programs).
Apologies if it sounded like I demanded a “cool new language enhancement” solution to this issue. On the contrary; if it were just documented that either fs::File [1] or the io::Write trait [2] could silently lose data with no error codes when dropped, that would be one such sufficient solution.
Perhaps I misread this documentation; if so, I don’t think I’d be alone here. I don’t see any particular mention that dropping an fs::File could lead to data loss, and I had generally assumed major edge cases like ‘data loss from a file system library’ would be documented.
It falls out of general principle; destructors may not be called. That said, I would happily accept a PR to make this explicit. Even an issue would be nice!
The alternative to closing the file in the destructor is leaking OS resources in the event of unexpected control flow destroying the file object. I fail to see how that is preferable.
Not in Rust, and not in any language where you can put resources in objects with shared ownership (i.e. any remotely popular general purpose language).
Throwing exceptions isn't a particularly good solution either, for the same reason. Exceptions are hard to reliably handle when you can't easily reason about where they will be thrown from.
It's not resolved because it's (a) tricky and (b) is really just a minor question about naming conventions and mutability.
I'm quite comfortable stating that non-lexical lifetimes and async I/O, for instance, are far more important. The number of users who benefit from those two features are multiple orders of magnitude greater than the number of users who care about whether the official opinion of the guidelines subteam is that close() should take &self or &mut self. The Rust team would be doing a disservice to users if it focused on small issues like that—this isn't even a bug we're talking about, it's guidance around conventions!—instead of the biggest complaints that come up constantly.
As I mentioned in my other reply to Steve Klabnik, documenting this edge case would have been a sufficient “resolution” to the bug at hand.
I may call it a bug, and you may call it undocumented silent data loss behavior; either way, we’re talking about the same thing. Silent data loss from undocumented behavior is not good, wouldn’t you agree?
I certainly was not aware that drops in Rust could throw away potentially serious error codes. Now I have to go re-audit the correctness of all my Rust code that uses the file system (at the very least).
If the behavior was documented I would not consider this a bug. That said, perhaps I’m missing something in the documentarion — my apologies if thats the case — but I did just re-read the fs::File docs and see no mentions or precautions about potential data loss when a File is dropped.
I mean, I guess it's not documented as well as it could be, but the only alternative to dropping close errors would be panicking, and that would be significantly worse. Destructors aren't supposed to fail.
> the only alternative to dropping close errors would be panicking, and that would be significantly worse.
There are plenty of cases where you'd prefer an application to crash upon an unhanded write error, rather than silently losing data that could be highly important and irrecoverable.
(Of course, actually handling the errors is preferred above both.)
> Destructors aren't supposed to fail.
But isn't the whole point of this discussion is that the destructor of fs::File (and probably any other buffered IO writer) can and does fail in some cases?
> But isn't the whole point of this discussion is that the destructor of fs::File (and probably any other buffered IO writer) can and does fail in some cases?
And the choices to handle such failed destructors are: blockingly retry until the problem goes away or just plain ignore it. Either way you can't rely on destructors for persistence/durability in case of a crash or power loss.
> There are plenty of cases where you'd prefer an application to crash upon an unhanded write error, rather than silently losing data that could be highly important and irrecoverable.
Not in the programs I write. I grant that perhaps this should be configurable.
If you really care about reliability you implement transactional semantics on your output storage: i.e. writes are not globally visible until an explicit system wide atomic commit is preformed.
The destructor would instead be in charge to perform the rollback actions on an uncommitted transaction, if any. Rollback cannot fail and indeed the system must preserve integriy even if not performed as there is no guarantee that the process will not be killed externally.
Of course if you do not care about data integrity, swallowing errors in close is perfectly acceptable.
Edit: in general destructors should only be used to maintain the internal integrity of the process itself (freeing memory, clising fds, maintaining coherency of internal datastructures), not of external data or the whole system. It is fine to do external cleanup (removing temporary files, clearing committed transaction logs, unsubscribing from remote sources, releasing system wide locks etc), but shoud always be understood to be a best effort job.
A reliable system need to be able to continue in all circumstances (replying or rolling back transactions on restart, cleaning up leftover data, heartbeating and timing out on connections and subscriptions, using lock free algos or robust locks for memoryshared between processes, etc).
Python's "with" construct is analogous to the bracket pattern in Haskell that the article is talking about. It also works in the nested case in the presence of exceptions. Furthermore, the issue that Michael has with the bracket pattern in Haskell can also happen in Python.
True, but in Python the coding mistake would stand out much more because the with block is syntax sugar - it does not look like regular function application, whereas in the Haskell example there is nothing to tell you that withMyResource is using the 'bracket pattern' (except by reading the src)
Also I guess in Haskell there is more expectation that the type system should prevent you from expressing runtime errors
I can see why you might think that, being built into the language, using 'with' in Python in a broken way would be easier to spot. However, having used both languages extensively, I can tell you that, at least for me, there's no discernible difference.
I think the reason for this is might be that, in Haskell, a function starting with 'with' is, by convention, using the bracket pattern and the way that you might use such a function would be very similar in structure to the Python way.
Something that is often said about C++ is that, you're only ever using 10% of the language, but that everyone uses a different 10% and it's true, but it's true of every language to differing degrees. Everyone has their own way of forming programs, just like everyone has their own slightly different style of playing chess, cooking or forming sentences.
When you have a well developed style, you will quickly spot any deviations from it. At that point, it doesn't matter if your style was forced on you by the language or whether it's just a convention that you use.
It's certainly true that Haskellers expect a lot from the type system, even compared to other static languages, let alone Python.
I only mean it's visibly more obvious, you have an indented block... what is the purpose of the indented block unless to say "do all your stuff with the resource _inside_ this block". Using the with block is very 'intentional' feeling.
I'm not very familiar with Haskell but it seems like you'd get used to the type system telling you everything you need to know. But in this case it doesn't. In Python world we talk about 'pythonic/unpythonic'... it seems like it's maybe quite unhaskellish to have to rely on a naming convention and remembering not to use the return value of the function?
I would guess that's why the article and many of the other comments here focused on how you could express this behaviour in Haskell's type system, where you'd expect it.
In short: type system > syntax sugar > naming convention
> I only mean it's visibly more obvious, you have an indented block...
Haskell is more similar than you realise, it's the difference between this:
withSomeResource $ \resource -> do
someFunctionOn resource
and this:
with some_resource() as resource:
some_function_on(resource)
> I'm not very familiar with Haskell but it seems like you'd get used to the type system telling you everything you need to know
As an outsider, you might expect a type-error to mean that you made a logic error, in practice it usually means you made a typo.
What happens is that the type system forces you to write things in a certain way. You internalize its rules and it moulds your style. You don't try random things until they stick, you write code expecting it to work and knowing why it should, just like you would in Python. It's just that more of your reasoning is being verified. "Verified" is the operative word here - the type system doesn't tell how to do anything.
> it seems like it's maybe quite unhaskellish to have to rely on a naming convention and remembering not to use the return value of the function?
The Python equivalent of the problem here would be:
current_resource = a_resource
with some_resource() as resource:
current_resource = resource
current_resource.some_method()
So it's not that using the return value of the withSomeResource function is a problem, it's the resource escaping from the scope where it is valid.
I think the crux of our discussion is about checked vs unchecked constraints.
When you work on (successful) large codebases, whether in a static or dynamically typed language, there are always rules about style (and I mean this in a broader way than how your code is laid out). For example, in large Python projects, there might be rules about when it is acceptable to monkey-patch. These rules make reasoning about the behaviour of these programs possible without having to read through everything.
Large Haskell projects also have these rules, but Haskellers like to enforce at least some of them using the type system. It takes effort to encode these rules in the type system and it is more difficult to write code that demonstrably follows the rules than implicitly follows them, but the reward for this effort is that it gives you some assurance that the rules are actually being followed everywhere.
For some rules this extra effort makes sense and other times it doesn't. The type system is just another way to communicate intent. Writing the best Haskell doesn't necessarily mean writing the most straight-jacketly typed Haskell, but it does give you that option. Beginners often fall into the trap of wanting to try out the new-and-shiny and making everything more strict than is helpful.
For one-man projects, there's really no advantage to Haskell over Python (with the caveat that you may not remember all of the intricacies of your code in six months and using Haskell you may have encoded more of your assumptions in the type system).
with some_resource() as resource:
some_function_on(resource)
Is that broken? If some_function_on saves the resource, yes. If it just temporarily uses it, no.
I don't think the claim that it's syntactically obvious in Python is correct. In both cases the typical syntax helps a little but it's easy to get wrong.
It is the case that "the typical syntax" is a little more enforced by Python-the-language.
I don't think Rust can notify on a failing destructor other than panic!ing. AFAIK the best you can do if you want to handle errors on close is to call `flush()` (which does return errors) before dropping the object. Of course that nullifies the benefits from RAII.
I don't know if there's an elegant way to solve this. If Rust had exception you could use that but then again in C++ it's often explicitly discouraged to throw in destructors because you could end up in a bad situation if you throw an exception while propagating an other one. How does Python's "with" handle that?
> I don't think Rust can notify on a failing destructor other than panic!ing.
Much as in C++, this is not really allowed: drop runs during panic unwinding, a panic during a panic will hard-abort the entire program.
> I don't know if there's an elegant way to solve this.
I don't really think there is. Maybe opt-in linear types could be added. That would be at the cost of convenience (the compiler would require explicitly closing every file and handling the result, you could not just forget about it and expect it to be closed) but it would fix the issue and would slightly reduce the holding lifetime of resources.
Furthermore, for convenience we could imagine a wrapper pointer converting a linear structure into an affine one.
> How does Python's "with" handle that?
You'll get the exception from `__exit__` chaining to whatever exception triggered it (if any). Exceptions are the normal error-handling mechanism of Python so it's not out of place.
>Much as in C++, this is not really allowed: drop runs during panic unwinding, a panic during a panic will hard-abort the entire program.
Right, I didn't really consider that a "drawback" because I'm in the camp that considers that panic! shouldn't unwind but actually abort the process here and there anyway. But you're right that if you rely on the default unwinding behavior panic!ing in destructors is a very bad idea.
> But you're right that if you rely on the default unwinding behavior
You do rely on the default unwinding behavior anyway at least for `cargo test`: the test framework depends on being able to catch the unwinds from `assert_eq!` and similar.
> I don't know if there's an elegant way to solve this.
It could take a callback. Then for any given file handle, if you don't care that the write failed you can ignore it; if you care but can't sensibly respond, you can panic; if you can sensibly respond you can do it inline or schedule work to be done somewhere with a longer lifetime than the file handle.
I do prefer the "with"/"try-with-resources" approach because it is explicit.
With RAII in C++ there's no visual difference between dumb data objects and objects like locks that are created and held on to mainly to cause implicit side effects.
In Rust this also prevents the compiler from dropping objects early - everything must be held until the end of its scope for the 0.1% of cases where you're RAII managing some externally visible resource. In those cases I would like the programmer to denote "The exact lifetime of this object is important", so the reader knows where to pay attention.
You can call std::mem::drop if you'd like to drop something early. That's the notation you're asking for.
Additionally, part of Rust's core ideas is that the compiler has your back with this kind of thing, so there's less need for comments that say "CAUTION HERE BE DRAGONS." Those things can still be useful for understanding details of your code, of course, but they aren't needed to ensure that things are memory safe. That's what the compiler is for!
I do use explicit drop() calls in my own code to call attention to drops with side effects, but it does not seem to be common practice.
My preferred semantics would have been early drops by default, and a must_drop annotation similar to must_use, to say that objects like RwLockReadGuard should be explicitly dropped or moved.
Nice article, but not sure that the changes it talks about are actually required for the suggested semantics. The `must_drop` annotation suggested could just be a lint rather than actually being encoded in the type system. I don't know if early drops is related or possible though.
Java has the try-with-resources statement, and C# has the using statement. They're alternative forms of the try statement and they're functionally equivalent to Python's with statement using contextlib.closing.
The error handling in close is platform specific, so you have to convert the file into a raw fd/handle and then pass it to the appropriate libc methods.
At least without pretending that everything's OK. Report to user and abort process would be appropriate for many cases. Continuing work with half-written data without ever noticing it is not appropriate.
The same way you would handle an error on fwrite() or fflush(). Any of them failing means that the data wasn't written correctly; what to do with that depends on the program.
Python's "with" clause, and the way it interacts with exceptions, is the only system I've seen that gets this right for the nested case.