But that's not how I remember it?

Matt 12:30
He that is not with me is against me; and he that gathereth not with me scattereth abroad.

 

It appears to be so at first. Consider this from "The Pulpit Commentary":

II. A seeming contradiction. The statement just quoted from the Gospel of St. Luke (Luke 9:50) appears to be at variance with another statement further on in the same Gospel, where, at the eleventh chapter and twenty-third verse, it is written, “He that is not with me is against me.” The discrepancy, however, is only apparent. In order to perceive this, we must consider the occasions on which the words recorded were respectively spoken; for, as our Lord and his apostles usually adapted their language to the occasion, we shall thus best learn the design with which each of those sentiments was uttered. Accordingly, we learn that some one not consorting with Christ or his apostles was, nevertheless, casting out devils in the Savior's name, and that John forbade him. Our Lord sets John right in the matter by saying, “Forbid him not;” that is, do not interfere with any who may be attempting anything good in my name. And then he assigns the reason; for “he that is not against us is for us;” he who is not directly opposed to us is rather to be regarded as on our side; he who is not preventing our progress may be looked upon, at least negatively, as promoting it. Just as is intimated by the Apostle Paul on a certain occasion, even though envy and strife should be the impelling motive, if Christ is preached his cause is advanced, and “I therein do rejoice.” So here we may fairly understand the words of the Master to mean — Whosoever this man may be, or whatever may be his object, he is weakening Satan's kingdom by casting out devils, and therefore, so far from being against me, he must be looked upon as an auxiliary in the great war against the great enemy of man. Besides, by such forbearance as I thus counsel, he may be drawn into closer and more effective co-operation against the common adversary. Such is the plain meaning of the passage before us. On the other hand, in the second passage, our Lord had been charged by the hostile, cavilling Pharisees with casting out devils by Beelzebub the prince of devils. This charge had called forth the rejoinder of our Lord, that “every kingdom divided against itself is brought to desolation.” Such would be the case if Satan cast out Satan. The only reasonable alternative was that the Savior was casting out devils by the Spirit of God, and so the kingdom of God had come unto them. He follows up this reply by a warning against lukewarmness and an exhortation to decision, that the crisis had come when men must choose sides, that they must elect to take part with God or with Satan. Neutrality was impossible. In view of two kingdoms so opposed, there was no possibility of belonging to both; nay, there was no middle ground between loyalty and rebellion. If not on the side of the Savior, he must be on the side of Satan; if not a subject of the former, he must be a slave of the latter, and so an enemy to the cause of Christ: “He that is not with me is against me.”
III. THE SAME SUBJECT VIEWED FROM A PRACTICAL STANDPOINT. The one text implies that men may take different roads to the same place, or reach the same point by different routes. This is true morally as well as geographically. It condemns the narrowness that refuses to tolerate want of uniformity, and commends forbearance towards all who in reality serve the same Master and seek the same object, viz. the glory of God, though their forms may be diverse, their modes of worship different, and even their creeds divergent in expression. The other text affirms that, in the natural and increasing conflict between good and evil, our hesitation to unite with the good is tantamount with adhesion to the evil. The one text does not insist on uniformity, the other inculcates unity. Again, conformity to the same standards is not an indispensable condition of Christianity, as we infer from the one text; but cordiality in embracing Christ and espousing his cause is of its very essence. We are taught by the one that there may be many folds, though there is but one flock; but by the other that, as there is but one Shepherd, union to him is indispensable to membership in his flock. Further, the one makes charity to others imperative, provided they have the same great end in view, however divergent the means adopted for its attainment; the other requires of us decision for ourselves in seeking that end. — J.J.G.
Verses 42-50. Parallel passage: Matthew 18:6-9. —
Pulpit Commentary, The - The Pulpit Commentary – Volume 16: Mark and Luke.