For (1), if you’re contemplating people switching to other channels because your fee is too high, then I think you immediately need something like valves or you just end up with Q(f) = \bar{q} for f \le \bar{f} and Q(f) = 0 otherwise, with \bar{q}, \bar{f} being constants that are determined by what the rest of the network is doing, rather than your decisions.
I don’t think the details of an actual function is useful beyond simple examples though; in practice it’s unknowable (and likely changes over time as lightning adoption changes and other channels are created/closed), and instead you’d be doing point measurements of Q(f) vs Q(f+\delta) (by directly manipulating your channel’s fee and measuring change in traffic) or just estimating values (perhaps inferring from what other channels are doing, perhaps as feedback from the impact of what your valves are doing), and then just bumping your fee by \delta if a\cdot Q(f) < (a+\delta)Q(f+\delta).
Note that the partial derivative equations are general, and don’t depend on Q(f) being linear.
There’s no particular problem with negative fees here: negative fees with positive traffic (or positive fees with “negative traffic”) give a\cdot Q(a+b) < 0 but that’s a worse result than just setting a=0 and receiving 0 = 0 \cdot Q(0+b) income, so isn’t a global maximum. Provided Q(f) is non-increasing, you also won’t get trapped at a negative local maxima if you’re just doing point-wise optimisation, I think.
If you’re only looking at a single channel, negative fees are only interesting for rebalancing, but that extends your utility function beyond “make the most profit”, which isn’t considered here.
For (2), in this scenario Alice and Bob are forwarding payments, so the source and destination are external, and the route is predetermined by the sender. Senders can probably be expected to naturally generate some sort of non-increasing demand curve for any given channel; though as above, I expect something like valves is needed for that curve to end up being smooth/differentiable.
I think once you have something of that nature, you can probably imagine modelling all lightning payments across all channels simultaneously, which will give you Q values for each channel; then tweaking fee values for a channel shows you how Q changes for that channel. Calculating a global optimum or a nash equilibrium with that much interaction is probably pretty hard though?
I’m not sure it makes sense to think too hard about inbound vs outbound relationships – there are n^2-n such relationships, and you only have 2n knobs to twiddle (inbound/outbound fees for each channel$, so you’re fundamentally limited in how well you can optimise things once you have more than 3 channels, and if you’re successful at price discrimination, that just encourages the person being discriminated against to open a channel with someone else to route around you. I don’t think I can even come up with a way of using it for price discrimination – if you want to make it cheaper for X to pay Z through you than for Y to pay Z through you, why not lower the outbound cost to Z for everyone, but raise the inbound cost for Y to compensate, maintaining positive fees everywhere?
But the main point of this was to demonstrate that (a) you can get disagreement on what the optimal fee rate is between channel partners, so a fixed split enforced at the protocol level doesn’t make sense, and (b) at least with sufficient knowledge about the state of the world, pure self-interest can result in a stable and fair outcome.